Skip to content

Expand organization metrics coverage - #44

Open
jorgesancha wants to merge 5 commits into
mainfrom
feat/expand-organization-metrics
Open

Expand organization metrics coverage#44
jorgesancha wants to merge 5 commits into
mainfrom
feat/expand-organization-metrics

Conversation

@jorgesancha

@jorgesancha jorgesancha commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

  • add Prometheus endpoints for cluster, data transfer, Sink, LLM usage, backup, and Private Link organization Service Data Sources
  • expose current cluster values and latest backup state as gauges, and cumulative usage values as counters
  • include the new endpoints in organization_metrics, with stable organization/workspace labels
  • sort every endpoint by metric name as required by Tinybird's Prometheus serializer
  • add Grafana and Datadog overview panels for every newly covered Service Data Source
  • document the expanded coverage and individual scrape targets

Validation

  • tb --cloud deploy --check
  • deployed endpoint smoke tests for all six new endpoints and organization_metrics
  • live .prometheus response verified with 51 HELP/TYPE metric families
  • metric invariant checks: unique series, valid metric names, and finite numeric values
  • dashboard JSON validation: unique IDs and non-overlapping appended layouts
  • git diff --check

CI note

The repository workflow's Tinybird Local build and tests pass. I changed repository secrets to point to this workspace as the previous ones were failing: https://cloud.tinybird.co/gcp/europe-west2/organization_metrics_sancha_test

@pei-tinybird

pei-tinybird commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

I found and fixed three issues:

  • cluster.prometheus returned HTTP 400 because metric names were LowCardinality(String). They are now explicitly cast to String.
  • Ingestion exposed duplicate Prometheus series because it grouped by pipe_id but labeled by pipe_name. It now aggregates by the exported labels.
  • Two gauges used reserved Prometheus suffixes. They were renamed to cluster_os_memory_bytes and jobs_in_window, including dashboard references.

All endpoints now return HTTP 200, and the combined output passes promtool with no duplicate or invalid metrics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pei-tinybird pei-tinybird pei-tinybird approved these changes
@ienva ienva Awaiting requested review from ienva

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jorgesancha @pei-tinybird